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AUTORECRESS l VE  SPECTRAL  ESTIMATION,  L»w. 
SPECTRAL  SMOOTHING,  AND  ENTROPY1 


Emanuel  Parzen 
Institute  of  Statistics 
Texas  A&M  University 
College  Station,  TX  7784J 


Abstract 

Spectral  estimation  Is  motivated  by  Information 
divergence  distance.  Two  methods  of  spectral  estima¬ 
tion  are  developed  in  this  paper:  autoregressive 
spectral  estimation  (section  3)  and  log  spectral 
kernel  estimation  (section  4).  They  are  motivated  as 
parametric  and  non- parametric  estimators  which  mini¬ 
mize  "entropy"  or  "information  divergence"  distances 
between  raw  and  fitted  spectral  densities.  The  role 
of  entropy  concepts  in  the  statistical  estimation  of 
spectral  densities  (section  2)  is  explained  by  con¬ 
trasting  it  with  the  role  of  entropy  concepts  In  pro¬ 
bability  density  estimation  (section  1).  Adaptive 
procedures  for  forming,  and  combining,  these  estima¬ 
tors  for  an  observed  time  series  are  provided  by 
order-determining  and  truncation  (half-power)  point 
determining  criteria,  which  are  described. 

1 .  The  role  of  entropy  concepts  in  statistical 
estimation  of  probability  density  functions. 

Let  X  be  a  continuous  random  variable,  and 

X, . X  a  random  sample  of  X  (consisting  of  indepen- 

1  n 

dent  random  variables  identically  distributed  as  X) . 
The  distribution  function  F(x),  —<*<•,  and  the  proba¬ 
bility  density  function  f(x),-“<x<«,  are  defined  by 

F(x)-Pr(X<x),  f(x)-F'(x)  . 

The  entropy  (or  Shannon  information)  of  X  le 
denoted  by  H(£)  and  la  defined  by 

H(f)-  £-log  f(x)  f(*)dx 
-Ef[log  f (X) ]  . 

All  observed  distributions  are  assumed  to  have  finite 
entropy . 

A  maximum  entropy  density  is  a  probability  den¬ 
sity  f(x)  determined  by  maximizing  H(f)  over  all  f 
satisfying  certain  constraints  (uaually  involving 
moments  of  f ) . 

Theorem  1A:  Three  important  densities,  and  their 
characterization  by  a  maxlmimi  entropy  principle  are: 
(1)  uniform  distribution  over  an  Interval  a  to  b 
maximizes  H(f)  over  the  constraint  that  f  Is  non-zero 
only  on  the  Interval  a  to  b;  (2)  exponential  distri¬ 
bution  with  mean  u  maximizes  H(f)  over  the  constraint 
that  f  is  non-zaro  only  for  x>0,  and  has  mean  u  ;  (3) 

normal  distribution  with  Man  a  and  variance  a2  maxi¬ 
mizes  H(f)  over  the  constraint  that  f  has  mean  u  and 
variance  o2 . 

The  maximum  entropy  principle  Is  a  probability 
modeling  principle  in  the  foregoing  examples.  It 
becomes  a  statistical  estimation  principle  (which  fits 
distributions  to  data)  when  the  constraints  on  f  are 
expressed  in  terms  of  settle  means  and  variances;  it 
la  than  similar  to  the  method  of  moments.  A  maximum 
entropy  density  estimator  f  can  be  expressed  in 
symbols:  . 

H(f)-max  H(f) 
f 

where  f  Is  constrained  to  have  certain  moments  equal 
to  the  corresponding  sample  moments. 

An  altsmatlve  (and,  we  believe,  more  general) 
statistical  estimation  principle  is  provided  by  the 

This  research  use  supported  by  the  Office  of  Naval 
Research  (Contract  NOOOU-81-MP-lOOOl ,  ARO  DAAG29- 
80-C0070). 


cross-entropy  H  ( g ;  f )  and  In  format.  Ion  divergence  I  ( g ;  f ) 
of  two  probability  density  functions  f(x)  jnd  g(x>. 
Define 

H(g;f)  -  /"  i-log  g(xWf(x)dx 

-Ef j-log  g(X) J; 

I(g;f)  •  r  (-log  *|^-if(x)dx 

"£  [-log 
V  108  f (X)  1 

Note  chat  H(f)»H(f;f).  Another  name  for  Information 
divergence  Is  Kullback-I.lebler  information  number.  A 
minimum  Information  divergence  density  g  is  an  approxi¬ 
mator  to  a  specified  density  f  determined  bv 

I(g;f)  •  min  I(g;£) 

8 

where  g  Is  constrained  to  belong  to  a  specified  para¬ 
metric  family  of  probability  denticles. 

Theorem  IB:  Three  Important  examples  of  minimum 
information  divergence  approximators  or  estimators 
are:  (1)  f  Is  assumed  to  be  positive  only  over  the 
Interval  a  to  b,  and  g  is  any  uniform  distribution; 
then  g  Is  the  uniform  distribution  over  a  to  b;  (2) 
f  Is  positive  only  for  x^a,  and  has  a  finite  mean  u. 
and  g  is  the  two  parameter  exponential  distribution; 
then  g  is  the  exponential  distribution  with  mean  u  and 
domain  (a,*);  (3)  f  has  finite  mean  u  and  variance 
j2,  and  g  Is  any  normal  distribution;  then  g  is 
N(u,o2). 

Theorem  IB  may  have  been  first  explicitly  formula¬ 
ted  by  Thiel  (1981),  although  It  Is  Implicitly  known 
through  the  equivalence  of  maximum  likelihood  estima¬ 
tion  with  minimum  Information  divergence  estimation. 

An  important  observation  by  Thiel  is  that  Theorem  IB 
can  be  used  Co  prove  Theorem  lA,  and  thus  avoid  the  use 
of  the  calculus  of  variations.  We  extend  this  observa¬ 
tion  to  spectral  density  estimation  in  section  2. 

A  minimum  Information  divergence  density  g  csn  be 
expressed  as  a  minimum  cross-enttopy  density: 

H(g; f )  •  min  H(g;f) 

8 

A  cross-entropy  can  be  defined  for  an  arbitrary 
(including  discrete)  distribution  function  F(x)  by 

H(g;F)  *  [*  (-log  g(x) idF (x) 

Therefore  a  minimum  Information  divergence  density  g 
can  be  defined  for  an  arbitrary  distribution  function 
F  by 

H(g;P)  •  min  H(g;F) 
g 

where  g  is  constrained  to  belong  to  a  specified  para¬ 
metric  family  of  probability  densities.  Theorem  IB  is 
true  for  this  definition. 

Consider  now  a  finite  sample  and  a 

parametric  model  f4(x)  for  the  true  probability  density 
f(x),  Indexed  by  parameters  )  which  one  would  like  to 
estimate  from  the  sample.  A  maximum  likelihood  esti¬ 
mator  of  9  Is  defined  as  the  parameter  values  t)  maxi¬ 
mizing 

(.('))  -  •  log  fn(X. . X  ) 

n  n  9  1  n 

n 


Let  F(x),  -*»•**••  ,  denote  the  sample  distribution 
defined  by 


1 


F(x)  •  fraction  of  X. . ■<  •  x  • 

!  n  - 

One  can  express 

l-n(0)  -  f  Ion  t^x)  dF(x) 

-  -  H(f0;F)  . 

Therefore  maximum  likelihood  parameter  estimators  9 
yield  minimum  Information  divergence  densities 

By  Introducing  t  to  denote  the  (symbolic)  sample 
probability  density  of  the  sample,  one  can  regard  6 
as  satisfying 

Kfgjf)  -  min  ICf9;f)  • 

A  re~ interpretation  of  maximum  likelihood  is  obtained 
by  rewriting  the  Information  divergence  In  terms  of 
quantile  functions  whose  role  In  statistical  infer** 
ence  Is  emphasized  by  Parzen  (1979). 

Introduce  the  sample  quantile  function 

q<u)  -  f~1(u)  . 


S(w,  .  r  e-2"lwvR(v) 
v*-® 

.. ,  .  a  -2*iwv 
t(w)  •  e  0(vj 

v®-“ 

The  spectral  distribution  function  Is  defined  by 
v 

F(v)  »  /  f(w')  dw'  ,  0  <  w  <  l  , 

0 

When  p(v)  Is  not  assumed  to  be  suamable,  there  always 
exists  a  spectral  distribution  function  F(w),  O^w^l , 
such  chat  ^ 

o(v)  •  /  s^ffiwvdF(w) 

0 

When  o ( v)  is  summable,  It  has  the  spectral  representa¬ 
tion  ^ 

o(v)  •  /  e2ffiWVf (w)dw  . 

0 


Its  derivative  q(u)  •  «}*(u)  satisfies 
q(u)  f(Q(u)>  -  1  . 


Define 


dA(u) 


fq(Q(u)) 

f(Q(u)) 


-£%«<«» 


where  F<j(x)  Is  the  distribution  function  with  density 
f„(x).  Hake  the  change  of  variable  u  •  P(x), 
x  -  ij(u)  to  obtain 

t(fe;f)  -  I  -log  d9<u)  du 

which  one  can  Interpret  as  a  measure  of  how  close  to 
a  uniform  density  la  dQ(u).  The  full  consequences 
of  this  interpretation  are  explored  elsewhere. 

It  should  be  noted  that  one  can  define  other 
measures  to  minimize  to  form  parameter  estimators: 
examples  are 

/0  (d^(u)-l|2du  , 

whose  minimisation  leads  to  'modified  chi-square" 
estimators,  and 


1 

/„  ( P|j(0(u) )-ul2du  . 

whose  minimization  leads  to  "minimum  distance  esti¬ 
mators". 

Information  divergence  Ls  the  measure  that  most 
readily  generalizes  to  stochastic  processes. 

It  should  be  noted  that  only  the  parameter 
•■MitioB  problem  le  efficiently  solved  by  minimizing 
I(f<j;0.  The  problem  of  goodness  of  fit  is 
solved  by  considering  the  sits  of  the  difference  from 
the  uniform  distribution  D(u)  -  u  of  Dg(u)"F0(Q(u) ) 
for  0  ■  *».  The  modal  Identification  problem  Is  to 
find  distribution  functions  #  such  Chet  P(<5(u))  is 
parsimoniously  not  slgnlf leant ly  different  from  the 
uniform  distribution  D(u)  -  u. 


2.  The  role  of  entropy  concepts  In  statistical  esti¬ 
mation  of  spectral  denelty  functions. 

Let  Y(t),  t  -  1,  ...»  T  be  a  sample  of  e  Gaus¬ 
sian  zero  mean  stationary  time  series  with  covariance 
function 

R(v)  -  E(Y(t)r(c+v)J,  V  •  0.  *1,  ±2 . 

end  correlation  function 

otv)  •  "  Corrjt (t)  ,Y( t+v)  ]  . 

We  assume  R(v)  and  (v)  are  absolutely  suanable, 
and  define  the  power  spectrum  S(v).  0  ^  w  *_  1,  and 
the  spectral  density  f(w),  0  ^  w  ^  1,  by 


A  stationary  Gaussian  time  aeries  is  called 
White  noise  if 

P(v)  •  0.  v  >  0  ; 
f(w)  -  1,  0  <  w  <  1  ; 

F(w)  -  w,  0  <  w  ^  1  . 

A  stationary  Gaussian  time  series  with  sismable 
correlation  function  and  lntegrable  log  spectral  den¬ 
sity  can  be  represented  In  terms  of  a  white  noise  time 
series  e(t)  representing  the  Innovations  [prediction 
errors  Yv(t)  -  Y(t)  -  Yu(t)  of  the  Infinite  memory  one- 
step  ahead  predictor  Yu(t)  of  Y(t)J.  The  AR(«*) ,  or 
infinite  order  autoregressive  representation,  is 

Y(t)+a,(l)Y(t-l)  ...+aa(n)Y(t-n)+...-  e(t)  . 

The  MA(») ,  or  Infinite  order  moving  representation,  la 
Y(t)-e(t)^s>(l)e(t-l)-fb-(2)c(t-2)+. . . 

A  finite  parameter  representation  is  an  ARMA(p,q)  of 
the  form 

Y(t)+e  (l)Y(t-l)+...+a  (p)Y(t-p) 

P  P 

-  *(t)+b  (l)e(t-lH. .  .+b  (q)e(t-q) 

9  9 

The  filter  relating  Y(t)  and  e(t)  Is  called  a 
whitening  filter.  Parameter  estimation  ls  the  theory 
of  estimation  of  tha  parameter*  of  ths  whitening  filter 
and  model  identification  Is  the  theory  of  estimation  of 
ths  structural  form  of  tha  whitening  filter.  To  deve¬ 
lop  approaches  to  parameter  estimation  for  s  random 
sample,  in  section  1  we  defined  the  following  concepts: 

Entropy  H(f)  , 

Maximum  entropy  density  f  , 

Cross-entropy  H(g;f)  , 

Information  divergence  l(g;f). 

Minimum  Information  divergence  density. 

Minimum  cross-entropy  density. 

Likelihood  of  s  sample  , 

Maximum  llkallhood  parameter  estimator. 

To  develop  approaches  to  estimation  of  ths  para¬ 
meters  0  of  a  parametric  modal  fe(w)  of  tha  spectral 
density  f(w)  of  a  stationary  zero  mean  Gaussian  time 
series  Y(t),  ve  develop  analogues  of  ths  foregoing 
concepts.  We  start  with  an  approximate  formula  for  ths 
likelihood  function 

f  io*  f9otu> . tom 

of  ths  time  series  sample.  Vis  assist  that  Y(t)  has 
been  divided  by  (R(0))H  so  that  it  can  be  considered 
to  have  variance  1,  and  its  covariance  function  equals 
Its  correlation  function. 


2 


The  first  step  in  analyzing  a  time  series  should 
be  to  compute  the  sample  correlation  function 
T-v  T 

i(v)  «  Z  Y(t  )Y (t+'v)  i  :  Y‘(t) 

t-l  t-1 

and  the  sample  spectral  density 

f(w)  -  .  :  Yft)e*2"lwti2  *  :  Y2(t) 

t-l  t-l 


-  -2*iVV 

i.  e 

!v|<T 


o(v)  . 


It  should  be  emphasized  that  in  practice  one 
should  consider  using  a  "data  window"  to  compute  f (w) , 
for  u  •  k/'Q,  k  •  0,1,..., Q-l,  by 

f(w)  -  jp(v)|2  ♦  ^ 

w  k-0  w 
T 

p(w)  -  Z  Y( t) K(—)exp (-Zwiwt ) 
t-l  X 


where  hero  f  denotes  the  true  probability  density  of 
the  sample,  and  f ^  Is  a  model  for  f.  It  should  be 
noted  that  we  are  using  the  notation  f  and  f9  with  a 
variety  of  meanings.  For  a  Gaussian  zero  mean  station¬ 
ary  time  series,  the  probability  density  of  the  sample 
is  specified  by  the  spectral  densities.  f(w)  of  the 
true  distribution  and  f9(w)  of  the  model.  We  continue 
to  denote  the  Information  divergence  by  IT  (f9;f)  but 
now  f  lndlcaten  a  spectral  density  rather  than  a  prob¬ 
ability  density.  Plnsker  (1963)  proves  a  formula  for 
If  (f^if)  In  the  limit  as  T  * ■  : 

lim  iT  ( f g ; f )  -  i(f0;f) 


where  1  f f ; f )  Is  the  Information  divergence  defined  as 
foLlows. 

For  two  spectral  Jensitles  f  and  g,  the  informa¬ 
tion  divergence  I(g;f),  cross-entropy  H(g;f),  and 
entropy  H(f)  are  defined: 


/ 


0 


,f(w) 

g(w) 


log 


t(w)  _ 

g(w) 


1) 


dw 


for  a  suitable  kernel  K(x)  (properties  of  windows  are 
discussed  In  Harris  (1978)).  In  addition  for  statis¬ 
tical  stability  one  should  then  slightly  smooth  f(w): 
(1)  compute  the  sample  correlation  function  by 

i  <M  k  .  k 

0(v)  m  —  Z  exp(2iri-v)  f(jj)  , 
v  k-0  w  w 

which  holds  for  0  <_  v  <_  Q-T  (and  therefore  one  may 
want  to  choose  Q  _>  2T) ;  (2)  compute  a  slightly 

smoothed  sample  spectral  density  by 

f(v)  -  Z  exp (-2t iw)  k(^)o(v) 

|  v  |  <T  M 

where  M  _>  T/2  and  k(u)  Is  a  suitable  kernel,  such  as 
the  Parzen  lag  window: 

k(u)  -  1  -  6u2  >  6|uj3  ,  |uj  '  0.5  , 

-  2(1  -  |u|)3  ,  0.5  <  |u|  <  1 

-  0  .  |u|  >  1  . 

Back  at  the  likelihood  ranch,  one  may  show  chat 
approximately 

-LjO)  -  j  log  2 if  +  H(f0;f) 

where  ^ 

H(f0;f)  -  \  [  (log  f  (w)  +  )dw 

0  9  ' 

This  formula  for  likelihood  shows  that  the  sam¬ 
ple  spectral  density  f(w)  is  a  sufficient  statistic 
for  a  time  series.  However,  it  Is  a  very  wiggly  func¬ 
tion  and  by  Itself  is  not  a  consistent  estimator  of 
f(w).  Estimators  ?(w)  of  f(w)  can  be  regarded  as 
"smoothings"  of  f(w),  but  the  basic  problem  la  how 
much  to  smooth. 

Another  aspect  of  the  likelihood  formula  is  Its 
justification  as  an  approximation.  To  those  misguided 
analysts  for  whom  max in ua  likelihood  provides  the 
ultimate  estimator  for  which  no  expense  should  be 
spared,  there  la  no  substitute  for  the  exact  llkail- 
hood  (which  of  course  Is  exact  only  If  the  model  being 
assumed  la  exactly  true).  Information  concepts  enter 
estimation  theory  when  one  recognizes  that  maximum 
likelihood  estimation  is  a  technical  device  for 
carrying  out  minimum  information  divergence  estima¬ 
tion.  Tha  Information  divergence  for  a  sample  Y(t), 
t-l,  ....  T,  le  defined  in  general  by 

fg(Y(l) . Y  (T) ) 

f(Y(l) .  ’ 


■H(g; f )  -  H(f ; f )  , 

H(g;f)-|  /  (log  g(w)  «■  dw  , 

H(f)-H(f;f)-4  /  Hog  f(w)  +  1)  dw  . 

"  0 

Since  u  -  log  u  -  l  »  0  for  all  u,  l  has  two  of  the 
properties  of  a  distance:  l(g;f)  0  ,  L(f;f)  ■  0  . 
However  I  does  not  satisfy  the  triangle  inequality. 

The  information  divergence  can  be  related  to  the 
t*2  log  spectral  density  distance 

L-L(f,g)  •  /  f Log  f(w)  -  log  g ( w) } 2  dw  , 

0 

using  the  fact  that  u  •  exp(log  u)  •  1  ♦  log  u  ♦  (>5) 
(log  u)2  .  When  f  and  g  are  "neighbors"  In  the  sense 
that  their  ratio  approximates  1, 

I(g;f)  -  j  L2L(f,*)  ; 

then  minimizing  l  Is  equivalent  to  minimizing  L.L. 

An  extensive  discussion  of  these  distances  Is  given  by 
Gray,  Buzo,  Cray,  and  Matsuyama  (1980). 

The  concepts  have  now  been  defined  to  state  some 
of  the  basic  fact9  of  parameter  estimation  theory. 

Maximum  likelihood  estimators  0  are  equivalent 
to  sample  minimum  cross-entropy  estimators  9  defined 

bv 

H(f,;f)  -  min  H(ffl;f) 

They  -'.an  be  regarded  as  estimators  of  the  population 
minima  cross-entropy  "parameters"  0* defined  by 

H(fg*;f)-  min  H< ;  f ) 

where  f  is  the  true  spectral  density. 

A  maximum  entropy  spectral  density  f  is  defined 
by 

H(f)  -  max  H(f) 
f 

where  f  La  constrained  to  satisfy  a  set  of  constraints 
of  the  form 
1 

/  f  (w)  f(w)  dw  -  C  .  J  •  l . M, 

0  J  1 

for  M  specified  functions  |».(w)  and  constants  C ^ ■ 

When  the  constraints  are  of  the  form 

/  «2wlw,f(w)  dw  -  ,.()),  j  «  1),  M . ♦  ■ 

0 


W»-  K(lo« 


Y(T» 


9 


>j 


1 


it  can  be  shown  that  f{w;  is  the  autoregressive  spec¬ 
tral  density  f  (w)  defined  as  follows: 
m 


• •  ,  2«lw. . - 

(e  )! 


where 


g  (*)  “Is-  .i(l)  *♦■...  *■  a  (m)z®  ; 
oi  at  m 

the  autoregressive  coefficients  a  (m) 

satisfy  normal  equations  (called  mYule-Walk5r 
equations) 

o 

J!  a  (k)p(k-J)  -  0,  1*1,  ....  o  , 
k-0  01 


where  a  (0)  »  1  ;  and 


m 

o(io 

"  k-0  ■ 

It  should  be  noted  that  from  a  sequence  o(v), 
v  "  0,  4-  1,  ...  one  can  quickly  compute  fa(v)  for  all 
successive  values  of  o  -  1,2,...  using  a  variety  of 
fast  algorithms  (see  Kallath  (1974)].  In  practice 
the  problem  la  to  determine  "optimal"  values  of  m. 

Some  important  properties  of  f  (w)  are: 

i  a 

(U 

/  e2"iw^  f  (w)dw  -  o(J),  j  -  0,  4-  1,  .. 

o  m 

.  ,4-  m  ; 

(2) 

tl  dw  -  L  J 

'  o  Vu) 

1 

(3) 

1 

/  log  f  (w)  dw  »  log  <J2 

o  ra  “ 

(4) 

H(f  ;f)  -  H(f  )  •  ~  dog  o2  4-  l)  ; 

m  m  2  "  m 

1 

■J2  •  /  !gn(z)|2  f(w)  dw 

(5) 

■  aln  /  |  ls-c.e^,TlW*-. .  .4-c  e^TTiwm|  2f  (w)  dv  ; 

c . . c  0  1  " 


(6)  gfBrz)hasall  its  roots  in  the  complex  plane  out¬ 
side  the  unit  circle; 

(7)  j^lra  log  •  log  j’; 

^  log  J2  •  /  log  f(w>  dw  ; 

0 

(9)  "  ft")  1 *  *  iB  differentiable  (the 

rate  of  convergence  of  £  (v)  depends  on  the  rate 
of  convergence  of  i2  to  a‘  ) ; 

(LOt  2I(fa;f)  -  log  >2  -  log  j*  •  0  as  m  -  » 

The  foregoing  facts  explain  why  autoregressive 
spectral  approximations.  Introduced  in  Parzen  (1966), 
(1969),  provide  powerful,  and  natural,  estimators  of 
an  unknown  spectral  density.  They  ere  generated  by 
the  "maximum  entropy  approach"  Introduced  by  Burg 
(1967).  However  the  Burg  algorithm  does  not  compute 
the  autoregressive  coefficients  ua(j)  and  the  Innova¬ 
tion  variances  <j£  by  the  Yule-Ualker  equations. 

Indeed  it  dome  not  compute  either  p(v)  or  ?(w).  It 
does  not  provide  insight  Into  how  to  Identify 
"optimal”  auto regressive  orders  a. 

•)ne  approach  to  defining  criteria  for  an  optimal 
order  m  is  to  examine  how  well  one  has  transformed  to 
white  noise  the  residual  series 


whose  specrral  density  is  glveu  by 
f  (w)  -  4  !g  (e‘ Blw) 1 2  f (w) 


.  Usl 

f  (w) 


A  "model  Identification"  determined  order  a  has  the 
property  that  the  spectral  distribution  function 

F  (u)-  ft <w')  dw'  ,  0  <  w  <  1  , 

m  a  m  -  - 

is  parsimoniously  not  significantly  different  from  the 
uniform  distribution  Fq(w)  -  w  representing  the  spec¬ 
tral  distribution  function  of  white  noise. 

In  deriving  autore  resslve  spectral  estimators 
or  approximators,  ve  ha\  so  far  developed  an  analog 
of  Theorem  1A,  by  stating  that  autoregression  provides 
maximum  entropy  estimators  subject  to  the  constraint 
that  certain  correlation  values  are  attained.  We 
prefer  an  analog  of  Theorem  IB,  which  states  that: 
the  minimum  information  divergence  parameter  estima¬ 
tors  of  an  autoregressive  modal  for  the  trus  spsctral 
density  are  provided  by  the  coefficients  which  satisfy 
the  Yule-Walker  equations. 

A  very  important  Tict  (that  may  not  be  widely 
known)  Is  that  the  maxima  entropy  properties  of  auto¬ 
regressive  spectral  densities  follow  from  their  mini¬ 
mum  information  divergence  properties,  using  the  fact 
that 

I(f  ;f)  -  H<£  >  -  H(f )  >  0  ; 

mm  — 

consequently  H(f)  H(fm).  Since  f  and  fm  satisfy 
the  constraint  that  their  first  m  correlations  equal 
specified  values,  the  entropy  H(f)  achieves  its  maxi¬ 
mum  value  at  f  -  f  . 

m 

3.  Autoregressive  spectral  estimators  and  order 
determining  criteria. 

Given  a  sample  Y(t),  c  -  1,  ...»  T,  there  are 
many  approaches  for  forming  autoregressive  spectral 
estimators,  because  (as  summarized  in  Parzen  (1981)] 
there  are  four  equivalent  ways  of  parametrizing  them: 
(A)  autoregressive  coefficients,  (B)  correlations, 
(C)  partial  correlations,  and  (D)  innovation  vari¬ 
ances.  Here  we  only  consider  starting  with  the  sample 
correlations  o(v).  Then  for  m  ■  1,  2,  ...  one  forma 

fm(x)  *  0*  I 8d<”)|  ‘  . 


where 

8aU)  •  1  +  a^d)  z  *  ...  4-  an(m)z®  ; 

che  sample  autoregressive  coefficients  a  (j)  satisfy 
the  sample  Yule-Walker  equations  98 

o 

Z  a  (k)  p(k-J)  -  0,  j  -  1 . a  . 

k-0  ® 

where  3^(0)  ■  1;  and  the  order  m  innovation  variance 
m 

c2  -  Z  a  (k)  p (k)  . 

■  k-0  ■ 

Define  by 

log  j2  •  /  log  ?(w)  dw  . 

0 

Then  a a  a  tends  to  T, 


f-u( t>  •  Y(t)  a^U)  Y(t-l)  4-... 4-  a 


2 £ ( f  ;f)  •  log  j2  -  log  j£ 

*  ■  — 


0 


) 


W«  desire  to  be  a  sequence  of  consistent  esti¬ 
mators  of  f  tn  the  sense  chat  if  one  chooses  m  as  .1 
suitable  function  of  T,  then  as  T  *  » 

Iff  ;f)  -  0  and  f  fw)  *  f(v)  , 

m  a 

in  probability,  or  with  probability  one,  or  tn  mean 
square.  The  firat  rigorous  proof  of  such  results  was 
given  by  Berk  (1974)  who  also  finds  the  asymptotic 
variance  of  fa(v),conf  irmiag  conjectures  in  Parzen(196!>. 

We  now  consider  the  problem  of  choosing  m  adap¬ 
tively  from  a  sample  of  siae  T.  Conceptually  one 
would  like  to  choose  n  to  minimize  One 

approach  to  such  a  procedure  is  given  by  Akalke  (1974) 
and  leads  to  an  order  determining  criterion  called  All*. 

The  Akalke  Information  criterion  computes  for 

m  •  1,2,  . . . 

AIC  (■)  -  log  5^  *  IS 

and  an  optimal  order  m  satisfying 

AIC(a)  -  min  AlC(m) 
m 

The  optimal  order  m  can  equal  0,  indicating  that 
the  time  series  is  white  noise.  The  value  of  AIC(O) 
can  be  adjusted  to  the  value  one  desires  for  the  pro¬ 
bability  of  rejecting  the  hypothesis  of  white  noise, 
when  In  fact  the  time  series  is  white  nolae.  We 
recommend 

AIC(O)  •  -  i  . 

Parzen  (1974),  (1977)  proposes  autoregressive 
order  determining  criteria,  called  CAT,  whose  founda¬ 
tions  are  different  from  those  of  A IC  but  which  usually 
lead  to  exactly  equivalent  orders  in  practice.  The 
time  series  model  Identification  problem  ia  to  esti¬ 
mate  the  infinite  autoregressive  transfer  function 


an  estimator  <tf  the  terms  In  J  which  depend  on 
thus  one  minimizes 


la  an  "unbiased"  estimator  of  .  At  n  ■  0,  we 
assign  CAT ( 0 )  -  -  (1  ♦  (1/D).  j 

It  should  be  noted  that  a  multiple  time  series 
version  of  CAT  is  given  In  Parzen  (1977). 

An  order  determining  criterion  which  is  consis¬ 
tent,  but  whose  behavior  in  practice  Is  controversial. 
Is  given  by  Hannan  and  <}uinn  (1979). 


Lon  Spectral  Kernel  Estimator  and  Cepstral 
Correlations 


The  approach  we  have  been  describing  for  forming 
"optimal"  estimators  f (w)  of  the  spectral  density  f(w) 
of  a  stationary  time  aeries  is  to  view  f (w)  as  a 
function  closest  to  ?(w)  in  a  distance  between  spec¬ 
tral  densities  given  bv  the  information  divergence 
I(f;f).  The  class  of  functions  from  which  i(v)  Is 
chosen  has  been  constrained  (or  specified)  parametri¬ 
cally,  In  the  sense  that  f(w>  la  of  the  form  fa(w), 
where  0  estimates  the  parameters  9  of  a  model  tQ(w) 
for  the  true  f(w). 

A  non-paramecric  constraint  is  to  Impose  a 
smoothness  measure  on  f  such  as  the  integral  square 
of  the  r-th  derivative  of  log  f(w),  denoted 

/  | (log  f (w)>  ^r) [ 2  dw  . 

0 


8,(*)  ■  1  ♦  a^d)  1  ♦  ...  ♦  a^Cn)  i”  ♦  ... 

by  a  sample  order  m  autoregressive  transfer  funcclon 
«»<*>•  To  evaluate  the  overall  mean  square  error  it 
Is  convenient  to  define 

JB  ■  E/  I  yr  -  yr  g„<«2nw)|zf<w)4v 

0  a  - 

which  can  be  shown  to  be  the  sum  of  a  vsrlencs  term 

E/  1  J!  *>U2'1'V  *_('2’lw)  I  2  «<w)dw 

0  a  m 


and  a  bias  term 


^  *.(«2"1W)|Z  f(w)dw 


One  then  seeks  to  choose  f  to  maximize  smoothness, 
while  minimizing  a  measure  of  distance  of  f  from  i. 
Wahba  (1980)  Introduces  the  estimation  distance 

1  .  1  ,  , 

/  |T*Qg  f(w)  -  log  f(w)l2dVfK/  |  (log  f(v))UM*dv 
0  0 

where  K  ia  a  penally  parameter  to  be  determined  adap¬ 
tively  by  the  data.  One  may  show  that  the  resulting 
estimators  of  g(w)  *»  log  f(v)  are  of  the  form,  called 
log  spectral  kernel  estimators, 

g(w)  -  (log  f(v»-  1  *xp(-2»rlw)k(~)  r(v) 

V" 

where 

1 

y (v)  -  /  exp  (2wtw)  log  f(v)  dw 
0 


One  can  show  that  tha  variance  term  is  approximately 


1 

T 


I 

)-l 


o 


-2 

3 


are  called  cepstral  corrslatlons.  end  the  kernel  k(x) 
Is  given  by  l compare  Parzen  (1958)) 


One  often  considers  only  two  values  for  r,  2  and  4. 

The  etetletlcal  properties  of  cepetrel  correla¬ 
tion  have  been  extensively  Investigated  by  fthanaall 
U974).  y  ! 

Sine •  Jt(£>  -  J  for  V  -  M,  y,  c«ll  M  ch,  "h»H 

pow.r"  i.g .  u,  ,„k  to  4d.ptiv.ly  d.t.ral n.  M  fro, 
th,  ,u,l«  to  alnimiz,  th,  risk  function 

1^  -  J(f;f)  -  E  L2L(f,f) 


issuaing  Log  f(w)  hat*  ■»  representation 

*(w)  -  log  f  (w)  •  exp(-2^iwv)  y{v)  . 
v»-<» 

Following  Wahba  (1980),  to  minimize  R^  one  minimizes 
an  estimator  of  It  of  che  form 

^  -  8(M)  -►  V(M.T) 

where  8(M)  and  V(M,T)  are  measures  of  bias  and  vari¬ 
ance  given  by 

BOO  -  Ar  i  (y(v))2v4r{l  +  (£)2rfz  . 

V(M.r)  ■  4  J“  (1-f  U2V1  du  . 

'  6  0 

A  closed  form  evaluation  of  the  Integral  in  V(M,T) 
can  be  obtained. 

5.  Iterated  spectral  estimation. 

Observed  time  series  do  not  usually  obey  the 
assumptions  made  in  the  foregoing  theory  that  Y(t)  is 
a  zero  mean  Gaussian  time  series  with  suamable  corre¬ 
lation  function.  We  call  such  a  cine  series  a  "short 
memory"  time  serves  (of  which  white  noise  Is  a  special 
case,  called  a  "no  memory"  time  series).  Otherwise 
the  time  series  is  called  "long  memory"  (Parzen (1982)1 

Autoregressive  spectral  estimators  are  especial¬ 
ly  suitable  for  matching  the  large  scale  osc illations 
of  the  spectral  density  of  a  long  memory  time  serLes. 
The  role  of  the  autoregressive  filter  is  then  to 
transform  che  time  series  to  a  short  memory  time 
series  (obtained  as  the  residuals  tw(t)  described  in 
section  21.  The  spectral  density  of  the  short  memory 
series*  which  can  be  regarded  as  the  fine  structure 
of  the  original  spectral  density,  can  be  estimated  by 
a  log  spectral  smoothing  estimator  as  well  as  by  an 
aucoregresslve  spectral  estimator.  Employing  two 
different  approaches  to  short  memory  spectral  estima¬ 
tion  Is  desirable  since  the  problem  of  spectral 
estimation  is  not  simply  a  problem  of  parameter 
estimation  but  is  also  one  of  model  Identification. 

Iterated  models  for  forecasting  long  memory  time 
series  are  used  by  Parzen  (1981)  under  the  name  of 
"ARARMA  models”  (see  Appendix  for  an  example). 
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APPENDIX:  Wolfer  Sunspot  Numbers  1846-1963. 

To  illustrate  the  application  of  some  of  the 
foregoing  ideas,  we  report  an  Iterated  autoregressive 
model  fitted  to  the  annual  time  series  Y(t)  of  Uolfer's 
sunspot  data  for  Che  years  1846-1963  (which  Is  a  sample 
of  length  T  »  118).  Our  ARARMA  taodel  fitting  algo¬ 
rithm  automatically  proposes  the  following  taodel  (which 
it  hopes  will  have  the  best  medium  range,  If  not  long 
range,  forecasting  capability) : 

YU)  -  Y (t)  -  .482  Y(t-10)  -  .554  Y(t-ll) 

YU)  -  1.009  YU-1>  +  .36 2  Y (t-2)  -  t(t) 

The  series  YU)  is  a  short  memory  time  series  to  which 
Y (t)  has  been  transformed  by  the  initial  autoregress¬ 
ion  on  YU).  A  a  an  estismcor  of  the  true  log  spectral 
density  f(w)  we  take,  up  to  a  normalizing  constant, 

' log  fy(v)  >  •  tlog  f y (w )  y  log  fm(w) 


where 

f,(wl  •  ’£ 1  *» 

is  the  autoregressive  spectral  density  corresponding 
to  the  transformation  from  Y(t)  to  YU). 

Figure  1  is  e  graph  of  the  Wolfer  sunapot  data 
(the  crosses  represent  the  one-step  ahead  predictors 
of  the  model  above).  Figure  2  graphs  the  Iterated 
autoregressive  log  spectre l  estimator  f log  fy(w))‘ 
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